At this point, you may be wondering, “Is there any way to keep both Type I and Type II error small?”

The answer is yes, and it involves power, which is described in the next section.

Grasping the power of a test

The power of a statistical test is the chance that it will come out statistically significant when

it should — that is, when the alternative hypothesis is really true. Power is a probability and is

very often expressed as a percentage. Beta (β) is the chance of getting a nonsignificant result

when the alternative hypothesis is true, so you see that power and β are related mathematically:

Power = 1 – β.

The power of any statistical test depends on several factors:

The α level you’ve established for the test — that is, the chance you’re willing to accept making a

Type I error (usually 0.05)

The actual magnitude of the effect in the population, relative to the amount of noise in the data

The size of your sample

Power, sample size, effect size relative to noise, and α level can’t all be varied independently. They’re

interrelated, because they’re connected and constrained by a mathematical relationship involving all

four quantities.

This relationship between power, sample size, effect size relative to noise, and α level is often very

complicated, and it can’t always be written down explicitly as a formula. But the relationship does

exist. As evidence of this, for any particular type of test, theoretically, you can determine any one of

the four quantities if you know the other three. So for each statistical test, there are four different ways

to do power calculations, with each way calculating one of the four quantities from arbitrarily

specified values of the other three. (We have more to say about this in Chapter 5, where we describe

practical issues that arise during the design of research studies.) In the following sections, we describe

the relationships between power, sample size, and effect size, and briefly review how you can perform

power calculations.

Power, sample size, and effect size relationships

The α level of a statistical test is usually set to 0.05 unless there are special considerations,

which we describe in Chapter 5. After you specify the value of α, you can display the relationship

between α and the other three variables — power, sample size, and effect size — in several

ways. The next three graphs show these relationships for the Student t test as an example, because

graphs for other statistical tests are generally similar to these:

Power versus sample size, for various effect sizes: For all statistical tests, power always

increases as the sample size increases, if the other variables including α level and effect size are

held constant. This relationship is illustrated in Figure 3-2. “Eff” is the effect size — the between-